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Moving On Up: Data and Experience Doing CMM-Based 
Software Process Improvement 


Abstract: An analysis of Software Process Assessment results from 48 
organizations undertaking 2 or more assessments is presented in this report. 
The analysis focuses on the time required to increase process maturity, as well 
as the most prevalent process issues faced by the 48 organizations. Results of 
the analysis are used to provide guidance to organizations embarking on a 
software process improvement effort. 


1 Background 

Ever since the late 1980s, software organizations have been striving to improve their software 
process maturity as a way to improve productivity and quality. The Capability Maturity Model®*^ 
for software (CMM®'^) defines the relationship between software process maturity and 
software project performance [Paulk 93a] [Paulk 93b]. In general, software process maturity 
serves as an indicator of the likely range of cost, schedule and quality results to be achieved 
by projects within a software organization. Specifically, the CMM defines 5 levels of process 
maturity that represent distinct plateaus of organizational performance. 


CMM and Capability Maturity Model are service marks of Carnegie Mellon University. 
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To achieve these levels of software process maturity, organizations must overcome a series 
of challenges by addressing sets of Key Process Areas (KPAs). The KPAs at each level 
together provide a capability within the organization to realize the improvements in 
performance characteristic of the specific maturity level. The KPAs of the CMM are depicted 
in Figure 1 below. 



Figure 1 Key Process Areas of the CMM 
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To assist organizations in their efforts to improve their software process maturity, the SEI 
developed a diagnostic method that allows organizations to assess their own software 
process maturity (or have an assessment conducted by a third party). The method^ identifies 
strengths and weaknesses in an organization’s software process relative to the KPAs of the 
CMM. The resulting pattern of process strengths and weaknesses determines an 
organizations level of software process maturity [Humphrey 87a] [Humphrey 87b] [Humphrey 
87c] [CBA Project 95]. 

The software community has been using SEI-developed methodology to improve their 
software process. As organizations have taken actions and made changes to improve, they 
have asked the SEI a variety of questions regarding the value of improving software process 
maturity. These include: What is the return on investment from CMM-based software process 
improvement? What have been the lessons learned by organizations that have improved? and 
How long does it take to improve? Previous work by the SEI has addressed some of these 
questions. Herbsleb et al reported on the return on investment for CMM-based software 
process improvement and included 5 case studies from organizations that have made 
improvements in their software processes [Herbsleb 94]. Kitson and Masters described the 
types of process issues detected by organizations beginning their initiatives with CMM-based 
software process improvement [Kitson 92]. Finally, the SEI publishes periodic updates of the 
Community Maturity Profile which shows the distribution of software organizations across the 
5 maturity levels of the CMM.^ 

In this report, we investigate the experiences of organizations following the CMM as a model 
for software process improvement (SPI). Specifically, we address the following questions: 

• how long does it take for an organization to move up a maturity level? 

• what are the process challenges that distinguish those who move from the 
initial level (level 1) to the repeatable level (level 2) and those who remain at 
the initial level? 


^ The original assessment method, Software Process Assessment (SPA) was developed prior to the release of 
the CMM. The SPA method has now been replaced by the new CMM Based Appraisal for Internal Process 
Improvement (CBA IPI). The original assessment method was based on [Humphrey 87c], and the CBA IPI is 
based on CMM vl.1. 

^ As of this writing, the latest update from April 1995 reports on 379 organizations. 
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2 Data Used in the Study 


Since the SEI developed the SPA method in 1987, it has been collecting assessment results 
from organizations using the method. The results include the maturity level of the organization, 
the identified process strengths and weaknesses, the organizational scope of the assessment, 
and the date the assessment was conducted. To address the questions posed above, we 
focused on organizations that have undergone multiple assessments. This allowed us to 
investigate the experiences and changes in individual organizations. From the database 
housing the assessment results, we extracted the data for 48 organizations that had 
conducted 2 or more assessments. In fact, as a group these organizations have conducted 
104 assessments. 


In order to understand whether this set of organizations was similar to the overall software 
community using CMM-based SPI, we compared it to the overall community on a few 
dimensions. We compared these reassessed organizations with the overall community in 
terms of time involved with CMM-based software process improvement, type of organization, 
and maturity level profile. 

The year of first assessment can be taken as an indication of when the organization began its 
CMM-based software process improvement effort. Figure 2 shows the year in which these 
(reassessed) organizations did their first assessment compared to the overall community, 
(note: the reassessed organizations are included in the figures for “overall”). 



379 organizations overall 
48 organizations reassessed 


H Overall ® Reassessed 


Figure 2 Year of First Assessment 
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In general, the organizations in this study started their SPI efforts earlier than the organizations 
in the overall community. For instance, approximately 73% of the reassessed organizations 
began their SPI efforts in 1991 or earlier compared to approximately 32% in the overall 
community. 

Figure 3 shows the types of organizations included in this study compared to the overall 
community. 



or In-house Contractor Fed Org. 


H Overall H Reassessed 


Figure 3 Organization Type 

Consistent with the early applications of the CMM and SPA, a majority of the organizations are 
DoD and federal government contractors. However, since the original development and 
applications of the CMM and SPA, more commercial and in-house software organizations 
have initiated programs for CMM-based SPI. 
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Finally, Figure 4 shows that these organizations have achieved a higher maturity profile 
distribution than that of the overall community. 


80% 

70% 

60% 

50% 

40% 

30% 

20 % 

10 % 

0% 



42% 



Initial Repeatable Defined 


0 . 6 % 

+---- 

Managed 


■ Overall ■ Reassessed 


0.3% 


Optimizing 




Figure 4 Relative Maturity Profile 

For instance, 42% of the organizations that have been reassessed have achieved the Defined 
Level of process maturity (level 3) as of the most recent assessment, while only 10% of the 
overall community have reached the Defined Level. Similarly, 73% of the overall community 
is at the Initial Level while only 25% of the reassessed organizations are currently at the Initial 
Level. 

In summary, the reassessed organizations differ from the overall software community. As a 
group, they adopted CMM-based SPI earlier, tend to include relatively more DoD and federal 
government contractors, and have progressed to a relatively higher software process maturity 
profile. In the following sections of the report, we analyze a subset of these reassessed 
organizations to answer some of the questions posed by those organizations involved with 
and considering the adoption of CMM-based SPI. 
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3 Time to Move Up a Maturity Level 


This part of the study was motivated by the customer-expressed need to have credible 
information about how long it takes for CMM-based SPI to have a noticeable impact on 
organizations. The most prevalent “conventional wisdom” has been that it takes 18 to 30 
months to improve a full maturity level but depending on the situation it could take much longer 
than this. The purpose of this analysis is to supplement conventional wisdom with empirical 
data. The uses for such data include 

• developing feasible SPI plans 

• setting sponsor expectations 

• setting expectations among the staff participating in a SPI effort 

In order to plan process improvement activities, it is important to understand how long it will 
take for noticeable differences to appear. Estimates for time required to achieve the next 
maturity level have been based on the judgment and experience of the decision maker. 
Furthermore, in a recent survey by Goldenson and Herbsleb, 86% of the process improvement 
champions surveyed agreed with the statement “process improvement is taking longer than 
we expected” [Goldenson 95]. 

Setting and maintaining the sponsorship of management has been recognized as an 
important contributor to the success of SPI programs [Humphrey 87a] [Daskalantonakis 94]. 
For senior managers, the level number has historically been the most widely recognized 
indicator. When goals like “Level 5 in ‘95” are set by senior managers, process improvement 
champions sometimes lack substantive information with which to suggest more feasible 
targets. 

While management sponsorship is vital, the ‘buy-in’ of technical staff is also crucial.^ The 
synergy developed during the assessment process is just as important as the sponsorship 
discussed above. Setting unrealistic expectations for the technical staff can squelch their 
motivation and create a backlash of cynicism that will be counterproductive in the end. 
Champions of SPI programs will benefit from providing technical staff with credible information 
about anticipated progress. 

To summarize, our goal in this section of the study is to report on the experience of early 
adopters of the CMM in order to provide input for better planning and to set expectations of 
management and practitioners alike. 


Sudlow, Bill, From Chaos to SEI Level 2 in 18 Months, Unpublished white paper, 1994. 
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3.1 Measuring Time to Move Up 

Change in an organization’s process maturity is a gradual transition. We would be hard 
pressed to identify a specific event or moment associated with an organization changing from 
level X to level X+1. But, for the purpose of this study, we use the day an assessment ends as 
the milestone. Thus, the conclusion of an organization’s first assessment is taken to be the 
time when CMM-based SPI began, and the conclusion of a subsequent assessment where 
the organization’s maturity level increased is taken as the milestone for an increase in the 
maturity level. This decision seems reasonable because the maturity level of an organization 
is not known until it is measured, and the final findings presentation is where this measurement 
is first reported. 


3.2 Findings of the Study 

On average, organizations moving from level 1 to level 2 did so in approximately 30 months 
(between the first assessment that found them at level 1 and the subsequent assessment that 
found them to be at level 2). Organizations moving from level 2 to level 3, on the other hand, 
did so with an average of 25 months. However, the averages fail to reveal interesting and 
important information found in the distributions of times to change a maturity level. 

Figure 5 is a graphical depiction called a “box plot,” which is used to show the distribution of 
number of months to move up a maturity level for the two groups. Please refer to Appendix A 
for a brief review of how to read the box plot. 


Figure 5 Group Differences in Time to Move Up 


Number of 
Months 
Between 
Assessments 



Group 1 

MLl to ML2 
19 Organizations 


Group 2 

ML2 to MLS 
17 Organizations 
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The “box plots” shown above represent the distribution of number of months between 
assessments for the two groups shown on the horizontal axis. The grey horizontal ban 
represents the conventional wisdom of 18 to 30 months. 

We can see from Figure 5 that the conventional wisdom is consistent with the experience of 
many of the organizations in this study. Among the organizations moving from level 1 to level 
2 (group 1), 37% of them did so within the 18 to 30 month interval. Among organizations 
moving from level 2 to level 3 (group 2), fully 50% of them did so within the 18 to 30 month 
interval. 

However, this reflects the experience of a minority of organizations studied here. Among the 
organizations in group 1, nearly 25% of them moved to level 2 in less than 18 months. In 
addition, many organizations in this group took significantly longer than 30 months to reach 
level 2. Table 1 (below) presents some summary statistics for the data reflected in Figure 5. 
As illustrated in Table 1,25% of this group took more than 35 months to reach level 2, with the 
most extreme case being 94 months.^ The distribution of times for group 2 is not as wide as 
group 1, but approximately 50% of organizations in this group moved to level 3 in more or less 
than the inten/al reflecting conventional wisdom (with 25% moving faster, and 25% moving 
slower). 

As reported earlier, the average number of months for these two groups differs by 5 months. 
However, the median number of months for the groups is equal (approximately 25 months). It 
is apparent that the differences in the means are due to the fact that many more organizations 
moving from level 1 to level 2 took longer to do so than their ‘higher maturity’ counterparts. 


Table 1 Summary Statistics for Time to Move up a Maturity Level 


Statistic 

Group 1 

Group 2 

Extreme Values or Outliers 

94 months 

48 months 

Largest Non-outlier Value 

58 months 

36 months 

75th Percentile 

35 months 

28 months 

Mean 

30 months 

25 months 

Median 

25 months 

25 months 

25th Percentile 

19 months 

18 months 

Smallest Value 

11 months 

12 months 


^ For clarification, the 75th percentile is the point on the distribution that divides the upper 25% and the lower 
75% of the organizations. 
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The most important finding of this particular analysis is that there is much more variability in 
the amount of time organizations take to move from level 1 to level 2 than there is in the time 
it takes to move from level 2 to level 3. Organizations planning their SPI programs will do well 
to keep this in mind as they set their goals. The change from level 1 to level 2 has the potential 
to take much longer than the conventional wisdom of “18 to 30 months.” Based on the 
experience of the 19 organizations studied here, 1 in 4 organizations that move from level 1 
to level 2 will take 3 years or more to do so. Fully half of the organizations studied here took 
more than 2 years to move up a maturity level, regardless of which level they were moving to. 
Finally, the wide variation in time to move up a maturity level suggests other factors should be 
considered and that predicting the time it takes to improve an organization is complex. 

It would seem reasonable to conclude that the experience of moving from level 1 to level 2 has 
an impact when the organization continues its progress by reaching level 3. The foundation 
for improvement, and lessons learned, from the first transition to the repeatable level may very 
well contribute to attaining the defined level more quickly. 
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4 Analysis of Software Process Assessment Findings 


The first SEI analysis of assessment findings was published by Kitson and Masters in 1992 
[Kitson 92]. These authors summarized the findings from 59 early applications of the SPA 
method. Kitson and Masters, in the abstract of their technical report, describe their study in the 
following way: 

It characterizes the software processes used by software managers and 
practitioners at the assessed sites and classifies issues identified during the 
assessments. The basis for the characterization and classification is a software 
process maturity model developed by the SEI. 

Now that the practice of Software Process Assessments, and the prevalence of CMM-based 
SPI has expanded considerably, the SEI has a larger pool of data upon which to draw. This 
enables additional analyses and the pursuit of more detailed research questions than was 
possible in 1992 when Kitson and Masters published their report. In the analysis presented 
below we will add to the information reported by Kitson and Masters, and focus on a 
comparison of organizations moving from level 1 to level 2 versus organizations remaining at 
level 1 for two successive assessments. 

4.1 Methodology 

This section of the report describes the methodology employed in the study. It covers how the 
assessment results were coded for analysis and the analytical approach used to produce the 
results. 


4.1.1 Definition of a “Finding” as the Unit of Analysis 

This study focuses on findings from software process assessments. The definition below was 
taken directly from the Lead Assessor’s Guide for CBAIPI: 

finding—An observation or collection of observations that have been accepted 
by the team as valid. A finding includes strengths, weaknesses, evidence of 
alternative practices, and evidence of non-applicable practices. A set of 
findings shouid be accurate, corroborated, and consistent within itself [Masters 
95 ]. 

The word “finding” is usually assumed to include strengths and weaknesses that are 
presented in the final presentation at the conclusion of an assessment. For the purpose of this 
study, we examined only those findings identified as weaknesses (or areas where process 
improvement was deemed necessary), any reference to “finding” in this report should be 
interpreted as “process weakness.” 

Based on this definition for the unit of analysis employed in this study, we were forced to 
exclude one of the 48 reassessed organizations from the remainder of the study.® 
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4.1.2 Mapping Findings to the CMM 

The lion’s share of the effort for this study was devoted to coding the CMM content identified 
in the 1485 findings under consideration. Our experience, and the experience of Kitson and 
Masters, told us that mapping findings to the CMM would not be a simple task. In order to 
identify issues and propose remedies for them, we conducted a reliability study of the mapping 
process. A description of the reliability study and our conclusions are presented in Appendix 
B. In the end, each finding was identified with respect to which KPA the finding addressed. In 
a small number of cases (29), a single finding was mapped to two KPAs. 

4.1.3 Analytical Approach 

We decided to analyze the data by identifying the presence or absence of findings related to 
specific KPAs, rather than counting the number of findings associated with each KPA for each 
organization in the data set. In our early analyses, we discovered that there were significant 
differences between assessments with respect to how many findings they reported. The 
number of findings per assessment ranged from 2 to 87 in our sample. 

While we would expect that within a given organization (using comparable assessment teams) 
variation in the number of findings would indicate variation in the number of significant process 
weaknesses, this is not likely to be true across organizations (and more importantly, across 
assessors). The grouping of process weaknesses (or the listing of more specific weaknesses 
by themselves) is determined by a trained team that has the goal of helping the organization 
to look objectively at their process and improve it. One assessment team might elect to group 
many detailed findings together into a single statement, while another assessment team might 
choose to present a ‘laundry-list’ of findings. These decisions made by the assessment team 
are based on their understanding of the needs of the organization, and how they expect the 
audience to respond at the final findings presentation. 

For these reasons, we decided to examine the presence or absence of findings mapped to 
each KPA, rather than counting the number of findings in each KPA for each assessment. As 
a result, when multiple findings address the same KPA, the occurrence of that KPA in the 
findings is counted only once for that set of findings. The percentages we report reflect 
percentages of organizations with one or more finding in each KPA, not the percentage of 
findings associated with each KPA. 


This organization had conducted three assessments, and their data were used in the analysis for time to move 
up a maturity level. The rest of their data were excluded from the remaining analyses because none of the 
findings reported could be construed as “process weaknesses” without substantial inferences on our part. The 
findings of these three assessments could best be described as “proposed solutions” rather than “process 
strengths” or “process weaknesses.” 
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4.2 Prevalence of Findings by KPA 

In this part of the report we present a brief comparison of our findings with those reported by 
Kitson and Masters [Kitson 92]. In general, the prevalence of issues associated with KPAs has 
not changed significantly since the Kitson and Masters report. The table below shows how the 
KPAs were rank-ordered in terms of prevalence in both studies. The lower ranks imply that 
more organizations had one or more finding in the associated KPA. Higher ranks imply the 
opposite — that fewer organizations reported weaknesses associated with the KPA. 
Therefore, we can see from the table that the most frequently occurring KPA in the findings of 
organizations included in both studies is Software Product Engineering (SPE). Conversely, 
the least frequently occurring KPA in both studies is Process Change Management (PCM). 


Table 2 Comparison to the 1992 Baseline 


Kitson & Masters 

Hayes & Zubrow 

Rank 

KPA 

Rank 

KPA 



Rank 

KPA 

1 

SPE 

10 

SCM 

1 

SPE 

10 

RM 

2 


D 

ISM 

2 

SPP 

n 

ISM 

3 


O 

QPM 

3 

TP 

D 

PR 

a 



TCM 

4 

SPTO 

n 

TCM 

a 


14 

PR 

5 

OPD 

D 

QPM 

6 

n 

15 

SQM 

6 

SQA 

15 

SQM 

7 

OPF 

16 

SSM 

7 

IC 

16 

SSM 

8 

SQA 

17 

DP 

8 

SCM 

17 

DP 

9 

RM 

18 

PCM 

9 

OPF 

18 

PCM 


Of the 18 key process areas in the CMM, 9 of them received the exact same ranking in the 
table above, while 7 differed by one rank, and only 2 differed by two ranks across both studies. 
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4.3 Differences Between Level 1 and Level 2 Organizations 

As depicted below in Figure 6, this section of the report focuses on 29 of the 47 organizations 
in the sample. Of these 29 organizations (that were all assessed at level 1 on their first 
assessment), 18 were assessed at level 2 on their second assessment while 11 were 
assessed at level 1 for a second time. In order to characterize the differences between level 
1 and level 2 organizations, we compare these two subsets of organizations. 



Figure 6 Moving From Level 1 to Level 2 Versus Remaining 

By looking at assessment findings from these two sets of organizations, we hope to shed light 
on what process weaknesses most clearly distinguish organizations at level 2 from their level 
1 counterparts. Knowledge about prevalent software process weaknesses allows 
organizations to take more effective actions to address SPI issues. Knowing that some areas 
are likely to be more difficult to address than others enables organizations to focus effort 
where it is most needed and be alert to those KPAs that have proved to be persistently difficult 
for others. 
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We started this analysis with a comparison of those organizations that moved to level 2 with 
those organizations remaining at level 1 in terms of the prevalence of level 2 findings on the 
first assessment. The figure below shows the percentage of organizations in each group with 
one or more finding in each of the level 2 KPAs on their first assessment. 



Figure 7 First Assessment Findings for Level 1 Organizations 

Several patterns of interest can be seen in the figure above. First, ail 29 organizations had one 
or more weakness in Software Project Planning (SPP), and over 80% had findings in Software 
Project Tracking and Oversight (SPTO). Second, among organizations that moved to level 2 
on the next assessment. Requirements Management (RM) and Software Configuration 
Management (SCM) findings were more prevalent. 

While the lack of statistical significance for these differences prevents us from making 
reasonable generalizations, the trends may suggest that the two “project management- 
related” KPAs at level 2 present a similar level of difficulty for process improvement, while the 
others differ in their level of difficulty. Our conjecture is that SCM and SQA can be thought of 
as “support functions” and RM as “managing the customer input.” This will be addressed 
further in the discussion that follows. 
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Next, we examined the differences between these two groups of organizations with respect to 
prevalence of level 2 KPA findings on the second assessment. 



Figure 8 Second Assessment Findings for Level 1 and Level 2 Organizations 

The contrast between the two groups is quite stark in the figure above. We see that 100% of 
the organizations assessed at level 1 on the second assessment had one or more finding in 
both Software Project Planning (SPP) as well as Software Project Tracking and Oversight 
(SPTO). Among organizations that moved to level 2 on their second assessment, on the other 
hand, the prevalence of SPP and SPTO weaknesses is significantly lower. While differences 
in prevalence of findings in each level 2 KPA exist for these two groups of organizations, the 
largest differences are found in SPP and SPTO, followed by Software Quality Assurance 
(SQA). 

Note that the prevalence of the non- “project management-related” KPAs among 
organizations that remained at level 1 is now increased as well. If our conjecture about the 
grouping of level 2 KPAs is correct, the project management KPAs are the most difficult, 
followed by the support functions, and finally managing customer input. The data shown in 
Figure 8 support the hypothesis that SPP and SPTO are much more difficult than any other 
level 2 KPAs, but the trends associated with the other level 2 KPAs is not sufficient for a 
conclusive study of this hypothesis. 
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Finally, the prevalence of findings in level 3 KPAs for these two groups is compared in the 
figure below. 



Figure 9 Second Assessment Findings for Level 1 and Level 2 Organizations 

The most notable difference between these two groups of organizations (with respect to level 
3 findings) can be seen in the prevalence of findings associated with Integrated Software 
Management (ISM). Less than 10% of the organizations remaining at level 1 on the second 
assessment reported findings in ISM, whereas over 70% of the organizations that reached 
level 2 reported findings in this KPA. 

ISM is often described as the level 3 version of the level 2 project management KPAs. The 
data shown in Figure 9 support the theory that organizations at level 1 are focusing on the level 
2 project management issues, whereas level 2 organizations are moving into the “higher 
maturity project management issues.” In addition, we know that much of ISM focuses on the 
project’s tailoring of the organization’s standard software process, and level 2 organizations 
are much more likely to be focusing on organization-wide processes than their level 1 
counterparts. 

Finally, it is interesting to note that the prevalence of findings associated with the Training 
Program KPA (TP) is high for both subsets depicted above. In fact, Kitson and Masters 
reported that 73% of the organizations they studied had reported findings in TP. In this study, 
this figure was 71%. Thus it seems that TP is viewed as an important KPA to focus on 
regardless of the maturity level of the organization. 
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5 Conclusions 


This final section of the report summarizes the practical lessons to be learned from this study. 
Interpretations of the results section are presented here for the reader to apply in their CMM- 
based SPI efforts. 

5.1 Time to Move Up a Maturity Level 

The experience of early adopters of CMM-based SPI (and assessments) suggest that it takes 
less time, on average, to move from level 2 to level 3 than it does to move from level 1 to level 
2. The data summarized in Figure 5 show that this difference in average time to move up is 
due mainly to the great variability in the amount of time organizations have taken to move from 
level 1 to level 2. We do not know at this time whether such marked differences will be 
observed when we compare these organizations with organizations moving to level 4 or 5. 

Improving process maturity by an entire maturity level in less than 18 months is a fairly difficult 
task. For the two groups studied here, approximately 1 in 4 organizations were able to 
accomplish this transition in less than 18 months. These conclusions are limited to levels 2 
and 3 at this time. When sufficient data from organizations moving to levels 4 and 5 are 
available we may find that moving to these higher maturity levels happens faster as a result 
of the foundation provided by the SPI experience at lower levels. 

Based on the median time to move up a maturity level for this sample, SPI champions and 
participants should not be concerned about taking more than 25 months to increase a full 
maturity level. Fully half of the early adopters studied here took more than 25 months to 
increase their process maturity by a full level. 

Published sources of information regarding the time it takes to improve a maturity level support 
the findings of this study. Published reports include organizations moving to level 2 in 1 year 
for Motorola’s Cellular Infrastructure Group [Daskalantonakis 94], 18 months for Procase 
Corporation,® approximately 2 years for Motorola’s Transmission Products [Johonson 94] and 
approximately 3 years for the Oklahoma City Air Logistics Center, which was clearly a 
successful SPI effort [Herbsleb 94]. 

Published reports of organizations moving to level 3 include: Ratheon, who report moving from 
level 1 to level 3 in approximately 4 years [Baatz 95]; Grumman Data, who reported moving to 
level 3 in a little over 2 years (ostensibly starting at level 1) [NIdiffer 95]; Hughes Aircraft: 
Software Engineering Division, who reported moving from level 2 to level 3 in approximately 
3 years [Humphrey 91]. 


® Sudlow, 1994. 
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5.2 Differences Between Level 1 and Level 2 Organizations 

The most obvious difference between level 1 and level 2 companies with respect to KPA- 
related weaknesses is seen in Figure 8. The prevalence of Software Project Planning and 
Software Project Tracking and Oversight in the process weaknesses of organizations is the 
most notable distinction between level 1 organizations and level 2 organizations. This 
observation is very consistent with the basic description of level 2 as ‘locusing on getting 
project management under control.” 

Several published case studies of software process improvement also point to the importance 
of project management. The findings of this study amplify some of the conclusions reached by 
authors of these case studies [Johnson 94]. In a 1994 survey by KPMG Management 
Consulting, respondents considering their experience with ‘runaway projects’ sited better 
project management as the key to preventing the problem [Computer Weekly 95\. Finally, the 
vital role of project management in software process was summarized by Russell H. Dewey 
in the following way: 

Though languages and tools are Important, the most important software 
productivity and quality improvements today are management, not technology, 
driven [Dewey 88], 
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5.3 Future Focus for Research 

The figure below shows the pattern of changes in maturity level for the set of organizations 
studied here. There are many interesting patterns of change in maturity level reflected in this 
figure, of which we chose to study a small part. 



Figure 10 Pattern of Maturity Level Changes 

The contrast between moving from level 1 to level 2 and staying at level 1 afforded us the most 
data for analysis. A similar contrast between organizations moving from level 2 to level 3 and 
organizations remaining at level 2 was not possible given the pattern above. In addition, the 
data available to date do not permit analysis of organizations at levels 4 and 5. There are 2 
organizations that reported decreases in their maturity level. Given this small sample size, we 
are unable to make meaningful generalizations about what process weaknesses are 
associated with this decline. 

The incidence of organizations “skipping” maturity levels is also of interest in the figure above. 
Four of the organizations in the sample moved from level 1 to level 3. One of these 
organizations, in fact, moved back to level 1 on their third assessment. As more data become 
available, we may be able to gain valuable Insights about how some organizations seem to 
make such drastic improvements in maturity level. The organizational characteristics 
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associated with fast or slow progress are of interest to many of our customer organizations. 
Questions like “what is the impact of the size of the organization?” or “how do people and 
technology contribute to process improvement success?” are two examples we have heard 
verbalized by SEI customers. 

Finally, the three organizations that moved from level 1 to 2 to 3 on three successive 
assessments reflect ‘steady progress’ which many organizations strive for. Contrasting the 
process issues of these continuously improving organizations with the weaknesses of others 
may reveal important insights about ordering process issues to resolve in order to ensure 
successful SPI upon which organizations can continue to build. Future studies may be able to 
address this very important issue, if the data reported to the SEI allow. 
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Appendix A A Word About Boxplots 

The purpose of this appendix is to provide the reader with a brief overview of how to interpret 
boxplots [Tukey 77]. 

Boxplots are often the best way to visually compare the dispersion (variability) of multiple data 
sets. By constructing box plots for several different groups on the same metric (as in Figure 5 
on page 8 of this document) one gets an impression of how the GROUPS differ, in a more 
complete way than a comparison of group MEANS. 

The figure below provides definitions for various parts of the boxplot. 
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Appendix B Reliability Study for Mapping Findings 

This appendix provides a brief description of the reliability study we conducted. The 
information provided here, for the most part, does not bear directly on the conclusions we draw 
in the paper, but helps the reader understand how the data were treated. 

Our motivation in conducting the reliability study was to ensure that the CMM content 
mappings were assigned in a consistent (repeatable) way. In the absence of reliability, it is 
nearly impossible to draw reasonable conclusions from the analyses. 

First, based partly on the earlier study conducted by Kitson and Masters, an initial set of coding 
rules was drafted and reviewed by the authors of this study. We knew from our assessment 
experience and conversations with Kitson and Masters that assigning CMM content codes to 
findings would be a non-trivial task. Our goal was to take advantage of the CMM structure in 
order to provide more detailed identification of the issues raised in assessment findings. 

Based on the initial coding rules, one of the authors assigned codes to each finding. These 
initial codes identified the 

• Key Process Area in which the process issue is discussed 

• Goal that would not be satisfied as a result of the process weakness 

• Common Feature that contains the process issue 

• Key Practice that addresses the process issue 

• difficulty in making the mapping (high, medium & low) 

If the finding addressed more than one CMM issue we allowed it to be mapped to two different 
sets of codes. These two sets of codes were identified as “primary mapping” and “secondary 
mapping.” 

Next we enlisted the help of two authors of the CMM to assign codes independent of our 
judgment. The CMM authors coded a random sample of 100 findings, and their codes were 
compared with those assigned by the authors of this study. 

Cur analysis of these codes revealed that there was sufficient agreement at the KPA level, but 
that codes assigned at lower levels of detail tend to differ both among the CMM authors and 
with the study author. 
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While investigating the reliability of the coding process we made the following observations: 

• 26% of the assessments were conducted prior to the publication of the CMM. 

• 94% of the assessments were carried out using the methodology known as 
Software Process Assessment. 

• 6% of the assessments were carried out using the new CBA-IPI method. 

• Discussions with the content experts lead us to conclude that looking at 
assessment findings without the context of participating in the assessment 
leaves much room for competing interpretations of a given finding. These 
competing interpretations seem equally defensible. 

• The finding area^ assigned by the assessment team is the most defensible 
code for CMM content, if the named finding area matches the name of a Key 
Process Area in the CMM. We found that 78% of the team-assigned finding 
area labels suggested (or named explicitly) specific KPAs in the CMM. 

• Reliable coding of findings to KPAs was possible. However, codes for goals, 
common features and key practices tended to differ substantially among the 
participants in the reliability study. 

Given this experience with the reliability study, we revised the coding rules in the following 
ways: 

• Rely on the team-assigned finding area unless there is a defensible reason 
not to (e.g., there is no obvious link to any of the KPAs in the CMM, based on 
the finding area alone). 

• Code the findings according to KPA only (as we did not expect sufficient 
reliability at lower levels). 

• Code the difficulty of mapping for each finding, and review the high difficulty 
mappings before proceeding. 

• For findings that pertain to more than 1 KPA, split the finding into 2, and 
identify each KPA. (In the end, this resulted in 29 additional findings for a total 
of 1514.) 

The findings were then coded following the revised rules, and the difficult findings were 
reviewed and discussed among the two authors of this study. 

All of the mapping was accomplished using a database created for this purpose. Each finding 
was reviewed multiple times before finalization, allowing us to review — for example— all 
findings that were mapped to a particular KPA or had a given level of mapping difficulty. 

Based on our knowledge of current assessment practice, we expect that new data generated 
by the CBA I PI method will allow more detailed analyses. Because of the more rigorous 
mapping of findings to CMM content, our hope is that future work in this area will focus on more 
detailed investigations at the level of Goals or even Key Practices. 


'■ By “finding area” we mean the process area identified by the team. This is typically the title of the slide on which 
the fin(jlngs is presente(j. 
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